Temporal Lift Pooling for Continuous Sign Language Recognition

AbstractPooling methods are necessities for modern neural networks increasing receptive fields and lowering down computational costs. However, commonly used hand-crafted pooling approaches, e.g., max average pooling, may not well preserve discriminative features. While many researchers have elaborately designed various variants in spatial domain to handle these limitations with much progress, the temporal aspect is rarely visited where directly applying or specialized be optimal. In this paper, we derive lift (TLP) from Lifting Scheme signal processing intelligently downsample features of different hierarchies. The factorizes input signals into sub-bands frequency, which can viewed as movement patterns. Our TLP a three-stage procedure, performs decomposition, component weighting information fusion generate refined downsized feature map. We select typical task long sequences, i.e. continuous sign language recognition (CSLR), our testbed verify effectiveness TLP. Experiments on two large-scale datasets show outperforms by large margin (1.5%) similar overhead. As robust extractor, exhibits great generalizability upon multiple backbones achieves new state-of-the-art results CSLR datasets. Visualizations further demonstrate mechanism correcting gloss borders. Code released (https://github.com/hulianyuyy/Temporal-Lift-Pooling). KeywordsLifting schemeContinuous recognitionTemporal